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Trevor R. Parmenter, 


Abstract 


The need for accoutttability in the area of the development 
of prograns for the pre-vocational preparation of the 
learning disabled is discussed. Various approaches | to 
evaluation are explored and a systems model ts proposed |. 
* as best meeting the needs of thosé who seek to evaluate the® 
. effectiveness of thetr programs, An example of how the 
model was used to evaluate a Work Experience program is, 
described, together with comments bet 9 made concerning tts 
utility. 


_ 4 
Introduction: In this period of minimal economic growth when marginal 
members of the workforce such as those with learning difficulties are 
particularly at risk vocationally it 1s crucial that we address our 
selves to the question of how effective are the various vocational and - 
pre- -vocational Progra which are mus nveonAg in our community. 


During more dropi tol times various. agencies such as Special schools 
had little difficulty’ in the initial placement of a large percentage 
of their school leavers in either sheltered or open employment, but the 
long term adjus tment of many of these initial successes has been thrown 
into question by follow-up surveys conducted in Victoria and N.S.W. 
(Limbrick, 1977). Andrews (1973) suggests that in thosé studies which 
have shown. that a relatively high percentage of mildly handicapped have 
been successfully placed in-employment, the criteria of "successful 
placement" may be open to different interpretations. It may be that 
many of the mildly handicapped too readily acquiesce with the decisions 
others make for them in work situations and their tacit conformity is 
erroneously taken as successful adjustment. This proposition finds 
some Support in the follow-up study of trainees from a Work Preparation 
Centre (Ward et al, 1978). Unfortunately, there is a dearth of inforn- 
ation on the long term vocational adjus tment of the handicapped; a 
situation that must be redressed if.we are to make more effective 
organizationel and instructional decisions during their adolescent. or 
earlier years. Despite years of research, indices of iat cone) LOSES. 
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sdeuats eae and enpToyabi ity have not yet BEEN derived 

The basic pasts of this” Kaper is that whenever Yocutfondl programs 

are planned evajuation should bean integral aspect, for as Wel }man 
and Moore (1975:1) observe, , - 


- Decisions regarding the establishment, maintenance, modi fic- 
ation, and continuation of our programs and activities should 
have a’ firm foundation of evidence of effectiveness. “The | 
educational community can no longer afford the luxury of: 
judging effectiveness from public popularity, legislative , 
authority, and unsupported professional judgment. 


A further assumption is that evaluation is -not ‘an end product, but a 
means to better program devélopment. Indeed, we are currently 
witnessing a quickening of society's questigning of educational 
outputs; reflected in Australia by public reactions. to the House of. 
Representatives Select Committee on Learning Difficulties - in Children 
and Adults. The need for a systematic approach to. the evaluation 
_ problem is therefore being accentuated by this demand: for accountability 
in education. , Brody (1976: 251) succinctly. BERS the case for 
accountability as follows: 


Instructional goalimtion ts Re tn both classroom and ~ 
institution. It tells us what Me .hape,accomplished and 

where we have failed. It provides information-so that both 
supportive and skeptical advoeates have concrete information 
to consider afd use. Most importantly, tt can be used to 

show parents, eliente, gnd taxpayers what we have accomplished 
with their time and mene 


Thus we can no longer be. Satisfied ‘with statements regarding inputs into 
a system as evidence of satisfactory provisions being made. 


ee 
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Aims_of Evaluation 


_-asstbly.ofe of the reasons for the chasm between research findings 
a“ “gad their subsequent use. is that the aims of program evaluation have not. 
been sufficiently clarified. There are various purposes for which eval- 

uation may be undertaken including: ae oo ; 


4 


s 


a) an analysis of _@-program's effectiveness; a, We 
* is 


b) \ the provision of data useful for making decisions: about 
a orogram's value; 


c) e facilitation.of program improvement; . " 


d)  or\all of the above. 


\ 


Types of Evaluation — ee . * 


” The particular aim or combination of aims that the program evaluator - 


: F 4 : 
has in mind will naturally determine the type or model of evaluation 
he will use. To this end there are a number of evaluation 
approaches and models. available with the toMy being the most 


cqmmon. . 


1. Formative vs. Summative Evaluation ae te. 


Scriven's (1967) distinction between "formative" or "summative" 
evaluation is wet] known and needs little elaboration here. Suffice to 
say that formative evaluation.is the process whereby data are 
used to develop a curriculum or ‘instructional unit to the stage ‘where 
it is ready to be used; whereas, summative evaluation is the ‘process of 
describing the end results or ‘effectiveness of a fully developed program 

‘package.. Brolin (1976: 248) adequately summarizes ,the. major character- 
istics of each as, follows: 
Formative evaluation’ (a) identifies deficiencies and potential 
improvements in existing curricula rather than/measyring 
the terminal effects of such curricyla, (b). locates weaknesses 
in student performance so teachers’can revise and \tmprove 
instructional materials and procedures, apg (c) helps make 
the instructional process intoa self»correcting mechanism. 
Conversely, swnmative evaluation determines what has been eee 


at the end of a project so’decisions can be made regarding the 
- repcacement of one curritéulwn by Gnaniee. an 


a Goal Attainment vs. Systen Model 


~~ Sehulberg er ae (1968) distinguish betyéen, two approaches which 


parallel those above. The goal attainment model which seeks to 
approx imate e experimental model evaluates the degree of success or 
failure encountered by the’ program in reaching its predetermined | 
objectives. Herein, however, may lie its greatest weakness for it is 
very difficult for an independent evaluator to obtain from-an organization a 
lid statément of program objectives. In’the extreme Etzioni (1960, . 
in Schulberg and Baker, 1968) suggests that organizational goals, ’ 

' @ he 


particularly pub Hes ones, have an iTlUsory quality in that ea 
may.‘never have been intended to be realized. 


In ‘terms of the. present discussion the very complex nature of . 

the. possible outcome variables of a vocational program are s ’ 
“such that it is | very difficult to objectify them al] completely, + 
’ particularly at the beginning of “an intervention process. i: 
Furthermore, the goal attainment mode 1 fails to recognize ; es 
adequately the subtle interplay among the multiplicity of variables Se 
which ‘impinge upan the process of vocational traiping. They include 
the background ‘and personal. attributes of the student or client; 

his physical, psychological, educational » occupational and social . 
attributes and the characteristics of. the agent or agencies which 
are operating to provide help and training. 


An alternative approach is to adopt a system model which assumes 
three basic stages; input, intervention and output. It further 
recognizes that there are at least four functions which should be 

‘ evident for.an organization to survive (Schulberg- and Baker, 1968): 


_ a) the achievement of goals and subgoals, a 
-b) the effective co- -ordigation of organizational sub-units, 
‘ ¢) the acquisition and maintenance of necessary resources,. 
4) the adaptation of the organization to the environment 
* and its own internal demands. ° 


Thede functions are acknowledged by system models such as those 
schematized by Wel Iman (1963) in his national study of guidance 
. (Fig. 1)’ and by Walls and Tseng (1976) in their paradigm of a 
“rehabilitation system (Fig, .2). . an - 


: es ‘ 
‘i. \ \ 
Insert Figs. 1 and 2 about here 
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These two models essentially have similar. components ‘in that. a struct-/ 
ure is provided for determining the relationship between major input 


and output variables, either via an experimental design to estimate 
cause and effect, or an association design to identify correlations 
for the generation of experimental hypotheses. The critical feature 
of a systems mgdel is that it allows for process evaluation in 
addition-to product or outcome evaluation. Having provision for © | 
continuous feedback of outcome results it allows’ for the me 
modification of the intervention pr®cess and provides for a better 
understanding of the relationship between one change and subseqient 
changes. Or as Wellman (1968:6) suggests,, 
‘the tmpact of change is hypothesized to be 
related to interaction effects as apposed 
to a chain of sequenttal events. 
In addition, the dynamic nature of the systems model provides *y 
a framework wherein the differéntiation among students and situations, 
as well as interactions among these’ variables, permits a degree of 
statistical control and interpretation that may lead to the kind. of 
differential conclusions required for definitive process evaluation, 
In other words, such an evaluative framework has more potential fon 
* programmatic utilization than does ‘the ‘summative or goal attainment 
“approach and affords a greater opportuni ty for individual needs to be \ 
“met. 


A Systems Model in Operation : 

The present author is currently engaged in two evaluation projects 

involving the vocational preparation of learning disabled adolescents. 

. In both cases a systems approach is being used. A Macquarie University 
Research team has been closely associated with the déve lopment and 

process evaluation of programs conducted within the Granville Work 
Preparation Centre. Here we have a‘fine example of: how both process 

_ and product evaluation work together effectively to provide constant feedback 
so that programs may be refined and adapted according to the changing 

needs of the client population and the rapidly changing economic, climate. 


\ ; » 
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However, I wish to dwell, in this paper, more fully upon the eval- 
uation of a Work Experience , Program at a Special School‘ for Mildly 
Intellectually Handicapped children. This school which attracted 

a grant from the Schools Innovations Program to set up a Work 
Experience Program invited the writer to participate in an 
augmented. evaluation of their project. Such an evaluation is 
characterized by the fact that the commissioned evaluator vorks 

with the project director in madvisory capacity, with the major 
-responsibility for the evaluation process itself resting with the 
project director and his team. This seemed an excellent opportuni ty 
for a systems model to be trialled and although at the time of | 
writing the full. report is not available, the following is an outline 
“of the procedures followed, : 


Under the terms of the Innovations Grant the school was able to 
*supplement its staffing resources to.allow the careers adviser more 
- time for field work, particularly the supervision of the work 
experience aspect of the program. In addition the school was able to 
utilize the resources of its local Technical College which trained 
‘students in a number of work skills. 


INPUTS 
These include the population served and its needs; together with the 
Situational variables that may impinge upon it. 


Procedure 


1). Data describing the population characteristics were gathered. % 


Wherever possible the variables were described in operational 
terms. These included - I.Q., vocational test results and 
recommendations, academic and social attainments, teacher ratings 
of social and emotional adjustment, degree of parental support 
and the level of vocational aspirations. held by subjects and 
their parents. Finally in this area, analysis and\documentation 
of the students' needs were made. 


4.2. The school's characteristics. and its value system were described, 
particularly the baseline conditions which operated ory to the 
implementation of the current program. ‘ 


. 1.3. The basic characteristics of the school's community were. 


2.0 


3.0 


experimental management and hence: are crucial to feasibility and 


* listed, together with the corporate gas and attitudes a 


that were held by the ‘various sections in this community — 

for the students, e.g. the parent body, the commercial /. ' 
employer groups, central and regional ies of Education, 
other Saver mIENE agencies, service clubs. 


1.4. A statenent of the current economic climate. 


PROCESS 


* 


Whereas- 4 inputs in this model may be seen as the statement of.a 

__ problem to be solved’(i.e. given this population and its vocational 
and social needs, how may we best meet them?) the process should be 
attempts at a solution. Here were stated (and quantified wherever. 
appropriate and possible) the program's objectives, the Strategies 
adopted, the resources made available, and the dynamics of the system. 
These are primary independent variables that can be subject to 


' 
transportability decisions for similar programs in other settings. 


Procedure 


2.1. Objectives were stated jmnerever practicable in operational terms of 


what a student will learn in the various areas of activity. - viz. 
\ prevocational, vocatjonal and technical. 


2.2. ~° Techniques and strategies employed were jaan Pisa: e.g. teaching ‘ 


procedures, special programs, group organization. | 


ae Resources were identified. These included additional staff, 


facilities (e.g. technical education), community involvement and 
equipment. 58 + ‘oe 


2.4. The dynamics of the organization of the process variables were 
monitored and described. Of particular relevance was the response of 
the organization to feedback from the ongoing evaluation of outcomes. 


OUTCOMES ; 

As far as possible objective assessments were made of the s tated 
objectives of the program. The instrumentation varied from subjective 
rating scales to standardized test batteries. An attempt was made to ° 
objectify evidence for value statements which were made. 


J ’ 


Procedure | 


a 
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3.3 


3.4 


3.5 


A framework was set up to enable a continuous criterion 
referenced evaluation of the students’ performance in: 


is the academic areas of the program. 

ii. the motor skills area of the program. 

iii. the work skills training area of the program. 
iv. social/prevocational program. 


Rating scales were developed to monitor the students’ progress 
in the personal and social development areas. Ratings were 
made by teachers, employers and parents. 


Instruments were developed which gathered data from teachers, 
employers, students and parents. concerning their attitudes 
to the program generally as well as to specific sections of it. 


In order to assemble-some local data for the commencement of what 


could be an Australian revision, two U.S. instruments; The 


Social and Prevocational Information Battery, Halpern et al 
(1975) and the Self Concept of Ability ‘as a Worker Scale, 

Burke and Sellin (1972) were administered to the subjects 
participating in the program. The latter scale was also administ- 
ered to a control sample of slow learners at a neighbouring high 
school. Preliminary results indicate that the Special School ‘ 
sample has a significantly higher self concept of their abitity 

as workers, but this result should be interpreted cautiously as 
our work at the Work Preparation Centre suggests that some 
learning disabled adolescents have an unrealistic concept of their 
abilities. 


, 


At the conclusion of.the program an analysis was made of the 
predictive value of certain pre-program assessments (e.g. Vocat-: 
ional guidance reports) for subsequent performance in the program 
a$ “revealed by the various outcome data. : 


Advantages of A Systems Model 


Experiences gained from this study and our work at Granville sugges t 
the following advantages of a systems approach: 


a) in contrast to other forms of evaluation it reduces the 
. threatening nature of evaluation and engenders a co- operative 
spirit rather than a defensive one. 


b) it is more client oriented. a 
c) it does not ask the question did it work, but rather why did it work? 


d) fail/safe mechanisms are built in to minimi e possibility of 
' failure. 


~ 


e) Ft allows for more effective replications; that is, it affords a 
greater opportunity for research to be translated into practice. ° 


f) ! it encourages longitudinal studies wherein both the quantitative as 
well as the qualitative aspects of a person's vocational, personal 
“and social adjustment may be more adequately determined, providing a 
more sensitive metric of the program's utility a 


-g) ih contrast to the goal-attainment model which is cdacerned with the 
degree of success in reaching a specific objective, the systems model 
establishes the degree to which the organization realizes its goals 
under a given set of conditions. \ 


h) it maximizes the probabilities that there will be a greater correspond- 
ence between job requirements and the individual's requirements. 


Changing Social Values and Economic Opportunites 


In the context of the current social and econdmic climate it is an ineluctable 
fact that our-population of children is under threat, witnessed by the tacit 
acceptance by many of the recently stated views of leading scientists such as 

Sir Macfarlane Burnet. Luckily; we still have apostlés of hope, such as 

Wolf Wolfensberger, Burton Blatt and Jean Vanier. On the economic scene, too, 

we are witnessing a diminution of the positive discrimination which began to 
be afforded to the handicapped under more buoyant times. 


10. 


It is, therefore, essential for those of us working in the area of the learning 
disabled to redouble our efforts*to develop programs which will stand up to 
rigorous scrutiny and not be gullibly seduced by what is ephemeral ly 
fashionable. No longer ‘can we afford to operate programs, be they- vocational 
or otherwise, in ana priori way, under. the assumption that what we “are doing 
is logical, reasonable and self-evident. In this period of static growth and 
limited resources the home, the school and the community must co-operate in 
ensuring that the cost benefits and cost effectiveness of all programs are of 


the highest order. 
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